You have two options for creating a custom group. You can select references and then add them to a new group or you can create an empty group and then add references to it.
To create a custom group:
Highlight the name of the group set to contain the new group.
Use one of these options:
From the Groups menu, select Create Group.
Control+click Right-click on the group set name to display a context-sensitive menu and select Create Group.
Select references in a reference list, then go to the Groups menu and select Add References To > Create Group.
Enter a group name that is anywhere from 1 to 255 characters in length. There are no restrictions on the characters you can use in a group name. It is possible to give two different groups the same name.
Add references to the new group. See Adding References to a Custom Group.
You are limited to 5,000 custom and smart groups per library. Once you reach that limit, you must delete a group before you can add a new one.
If you do not have a group set selected when you create a new group, the new group will be added to the nearest group set.